[New Integration] Initial commit of EfficientIP Intergration#18505
Open
jasperklaren wants to merge 7 commits intoelastic:mainfrom
Open
[New Integration] Initial commit of EfficientIP Intergration#18505jasperklaren wants to merge 7 commits intoelastic:mainfrom
jasperklaren wants to merge 7 commits intoelastic:mainfrom
Conversation
ReviewersBuildkite won't run for external contributors automatically; you need to add a comment:
NOTE: https://github.com/elastic/integrations/blob/main/.buildkite/pull-requests.json contains all those details. |
…tream/log/elasticsearch/ingest_pipeline/pipeline_dhcp.yml Co-authored-by: macroscopeapp[bot] <170038800+macroscopeapp[bot]@users.noreply.github.com>
…cluding manifest, sample events, documentation, and images.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed commit message
Title: Add EfficientIP DDI integration for DNS and DHCP log ingestion
WHAT:
This adds a new community Elastic integration for EfficientIP DDI (DNS, DHCP, IPAM)
appliances. The integration collects syslog messages over UDP and normalizes them
into ECS-compliant documents via a multi-stage ingest pipeline architecture:
A default router pipeline (default.yml) parses the syslog envelope using grok
(priority, timestamp, service name, PID), classifies messages by service name
(dhcpd/dhcpv6 → DHCP, named → DNS, httpd → AUDIT), and delegates to
the appropriate sub-pipeline. It also handles timezone-aware timestamp parsing,
GeoIP/ASN enrichment on client.ip, IP type conversion with error handling,
related.ip/related.hosts population, and recursive null/empty value cleanup
via a Painless script.
The DHCP sub-pipeline (pipeline_dhcp.yml) uses message-type-specific grok
processors gated by ctx.message.contains(...) checks to parse the full DHCP
lifecycle: DISCOVER, OFFER, REQUEST, ACK, NAK, RELEASE, EXPIRE, INFORM,
DECLINE, LEASEQUERY, and REFUSED events, plus DHCPv6 messages (Encapsulated
Solicit, Advertise NA, Relay-forward). Each message type has multiple grok
patterns ordered from most-specific to least-specific to handle variations in
EfficientIP's log format (optional client hostname, relay info, UID, transaction
ID, lease duration fields). MAC addresses are normalized (separator replacement
and uppercasing). All extracted IPs (client, relay, router, interface, link/peer
address) are converted to IP type and appended to related.ip.
The DNS sub-pipeline (pipeline_dns.yml) parses query logs and responses using
grok with custom pattern definitions (CLIENT, VIEW). A Painless script parses
multi-record DNS answer data (TTL, type, data tuples) into structured
dns.answers arrays. The registered_domain processor extracts the registered
domain from dns.question.name. Server IPs are enriched into related.ip and
query names into related.hosts.
Fields are mapped under efficient_ip.log.* with sub-groups for DHCP
(client_hostname, lease, relay, interface, trans_id, uid, etc.) and DNS
(category, answers, RPZ fields). ECS fields (client., server., dns.,
network., observer., process., event.*) are populated directly by the
pipelines.
The integration uses a single UDP input stream with configurable listen address,
port, timezone offset, and optional processors. Pipeline tests cover 6 DHCP and
7 DNS sample log lines with expected output validation.
WHY:
EfficientIP is a widely deployed DDI solution with no existing Elastic
integration. Organizations running EfficientIP lack a supported, turnkey way to
ingest and normalize their DNS/DHCP event logs into the Elastic Stack for
monitoring, threat detection, and compliance use cases (DNS query analysis, DHCP
lease tracking, network anomaly detection). This integration fills that gap as a
community contribution, with IPAM/AUDIT parsing planned for future iterations.
Checklist
changelog.ymlfile.Author's Checklist
How to test this PR locally
Related issues
Screenshots